1. Field of the Invention
The present invention relates to an image processing apparatus.
Priority is claimed on Japanese Patent Application No. 2012-222120, filed Oct. 4, 2012, the content of which is incorporated herein by reference.
2. Description of Related Art
In an imaging apparatus such as a still-image camera, a moving-image camera, a medical endoscope camera, or an industrial endoscope camera, an image processing apparatus of a pipeline configuration in which a plurality of image processing circuits, each of which performs image processing, are directly connected is well known. In the above-described image processing apparatus, as illustrated in FIG. 4A, a still image of one frame is divided into a plurality of blocks, and image processing is performed on each division block.
As illustrated in FIG. 4A, when a still image of one frame is divided into a plurality of blocks, the flow of image data to be processed within each division block is continuous. However, the flow of data between different blocks is not continuous (see FIG. 4B). Therefore, there is a need for a procedure to reset an image processing circuit every time processing of one block is completed and to reset the range of image data corresponding to the next block to be processed, or the like each time.
Thus, when an operation of the image processing circuit has been controlled every time each block has been processed, a period of time loss in which no image processing circuit operates occurs during processing of each block as described above. The loss time in which no image processing circuit operates affects the total processing time in a pipeline process of processing a still image of one frame.
For example, technology used to reduce time loss in processing between blocks in a pipeline process is disclosed in Japanese Unexamined Patent Application, First Publication No. 2010-176606. In this technology disclosed in Japanese Unexamined Patent Application, First Publication No. 2010-176606, each image processing circuit constituting a pipeline outputs an interrupt signal (process completion interrupt signal) indicating that a process by an image processing circuit has been completed to a sequencer which controls the overall pipeline process. Accordingly, every time the process completion interrupt signal is input from the image processing circuit, the sequencer individually resets the image processing circuit to change settings. Thereby, a timing at which the sequencer changes the settings of the image processing circuit is a timing at which a process of each image processing circuit for each block has been completed rather than a timing at which processing of a division block is started. In the technology of Japanese Unexamined Patent Application, First Publication No. 2010-176606, the sequencer sets the timing at which the setting of each image processing circuit is changed for every image processing circuit as described above, thereby reducing the time loss in processing between blocks and increasing the speed of the pipeline process for a still image of one frame.
In addition, in the image processing apparatus of the pipeline configuration, for an input/output of image data to be processed, a configuration in which image data stored in an external memory such as a dynamic random access memory (DRAM) connected to a common data bus (direct memory access (DMA) bus) through a burst transfer of DMA is accessed is adopted. In the input/output DMA unit provided in the image processing apparatus of such a configuration, for example, buffers of two sides are provided. Further, reset control of each buffer is divided and writing and reading of image data to and from each of the buffers of the two sides are alternately performed, and hence the processing time is shortened. For example, the output DMA unit shortens the processing time by performing an operation of pre-storing image data of the next block output from a previous-stage image processing circuit in the buffer of the other side while image data stored in the buffer of one side through the burst transfer of DMA is output.
Incidentally, there is an image processing apparatus which generates a plurality of images (hereinafter referred to as processed images) corresponding to the purposes for recording and displaying from image data of a still image of the same one frame among image processing apparatuses of the pipeline configuration. FIG. 5 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus of the related art. The image processing apparatus 100 of the related art illustrated in FIG. 5 is an example of an image processing apparatus that generates three processed images having the same angle of view and a different number of pixels (size).
In the image processing apparatus 100, a pipeline in which image processing circuits necessary to generate a first processed image are directly connected is configured. More specifically, the pipeline used to perform image processing is configured in the order of “DRAM 20→Input DMA unit 30→Luminance color converting unit 40→Noise reduction unit 50→Distortion correcting unit 60→Output DMA unit 81→DRAM 20.”
In addition, in the image processing apparatus 100, two pipelines used to generate other processed images by branching a data bus of a specific image processing circuit of the above-described pipeline are configured in order to generate processed images each having a different number of pixels. More specifically, a pipeline used to generate a second processed image in the order of “Resizing unit 71→Output DMA unit 82→DRAM 20” by branching a data bus of the distortion correcting unit 60 and a pipeline used to generate a third processed image in the order of “Resizing unit 72→Output DMA unit 83→DRAM 20” by branching the data bus of the distortion correcting unit 60 are configured.
In the image processing apparatus 100 of such a configuration, the setting and operation of each image processing unit constituting the pipeline are controlled by a sequencer that controls the overall pipeline process and image data flowing through the DMA bus 10 is controlled by a DMA bus arbiter.
Here, configurations of the resizing units 71 and 72 added to generate processed images each having a different number of pixels will be described. The resizing units 71 and 72 are only different in terms of the number of pixels of a processed image generated by a resizing process, and have the same configuration. Accordingly, in the following description, only the configuration of the resizing unit 71 will be described. FIG. 6 is a block diagram illustrating an example of a schematic configuration of the resizing unit 71 provided in the image processing apparatus 100 of the related art. The resizing unit 71 illustrated in FIG. 6 includes an input control unit 711, an input buffer 712, an output control unit 713, a resize operation processing unit 714, and a buffer full/empty monitoring unit 715.
The input buffer 712 temporarily stores image data output by the previous-stage distortion correcting unit 60 of the pipeline according to control from the input control unit 711, and outputs stored image data according to control from the output control unit 713 to the resize operation processing unit 714.
The buffer full/empty monitoring unit 715 monitors a storage capacity (buffer region) of the input buffer 712 based on a control signal from the distortion correcting unit 60 input to the input control unit 711 and a control signal from the subsequent-stage output DMA unit 82 input to the output control unit 713. Accordingly, the buffer Rill/empty monitoring unit 715 notifies the input control unit 711 of information about whether there is empty capacity in the input buffer 712, and notifies the output control unit 713 of information about whether image data capable of being output by performing a resizing process is aligned in the input buffer 712.
When a notification indicating that there is empty capacity in the input buffer 712 has been input from the buffer full/empty monitoring unit 715, the input control unit 711 outputs an image data output request (request) to the distortion correcting unit 60, and causes the input buffer 712 to temporarily store image data input from the distortion correcting unit 60 through a data bus.
The output control unit 713 receives an input of an image data output request from the output DMA unit 82, and causes the image data stored in the input buffer 712 to be output to the resize operation processing unit 714 when the notification indicating that image data capable of being output by performing the resizing process has been aligned has been input from the buffer full/empty monitoring unit 715. Accordingly, the output control unit 713 causes the output DMA unit 82 to write image data on which the resizing process has been performed by the resize operation processing unit 714.
The resize operation processing unit 714 performs the resizing process on the input image data at a predetermined resize ratio, and outputs image data after the resizing process.
In the image processing apparatus 100 of the related art, the resizing units 71 and 72 each having such a configuration are provided, so that a total of three processed images each having a different number of pixels and including a processed image in which distortion has been corrected by the distortion correcting unit 60 and two processed images obtained by resizing the processed image in which distortion has been corrected by the distortion correcting unit 60 are generated. That is, in the image processing apparatus 100 of the related art, a plurality of image processing circuits (resizing units and output DMA units) having the same configuration are provided according to the number of processed images to be generated, and a number of pipelines equal to the number required by the image processing circuits are configured. Accordingly, the processing time of image processing in the image processing apparatus 100 is shortened by operating the image processing circuits constituting the respective pipelines in parallel.
However, as in the image processing apparatus 100 of the related art, corresponding image processing circuits (resizing units and output DMA units in an example illustrated in FIG. 5) having the same configuration should be provided in the image processing apparatus 100 according to an increase in the number of processed images to be generated, that is, an increase in the number of branches of the data bus, in a method of configuring a necessary number of pipelines according to the number of processed images to be generated.
In addition, in the image processing apparatus 100 of the related art, each of the resizing units 71 and 72 does not necessarily constantly output a processed image generated by performing the resizing process. FIG. 7 is a timing chart illustrating an example of a timing at which image data on which each of the resizing units 71 and 72 provided in the image processing apparatus 100 of the related art has performed a resizing process is output.
In the timing chart illustrated in FIG. 7, an example of an operation period of the resizing process in which each of the resizing units 71 and 72 provided in the image processing apparatus 100 generates a resized processed image by resizing a processed image generated by the previous-stage distortion correcting unit 60 performing a distortion correction process on a still image of one frame to ½ or ¼ size and a timing at which image data after the resizing process is output to the subsequent-stage output DMA units 82 and 83 is illustrated.
A write signal I_WE illustrated in FIG. 7 indicates the timing at which the distortion correcting unit 60 outputs image data to each of the resizing units 71 and 72, that is, the timing at which image data after the distortion correction process is written to the input buffer 712 provided in each of the resizing units 71 and 72. In addition, a write signal O_WE71 illustrated in FIG. 7 indicates the timing at which the resizing unit 71 outputs image data to the output DMA unit 82, that is, the timing at which the resize operation processing unit 714 provided in the resizing unit 71 outputs image data after the resizing process to the output DMA unit 82 by performing the resizing process on image data after the distortion correction process temporarily stored in the input buffer 712. Likewise, a write signal O_WE72 illustrated in FIG. 7 indicates the timing at which the resizing unit 72 outputs image data after the resizing process performed by the resize operation processing unit 714 to the output DMA unit 83. In the timing chart illustrated in FIG. 7, image data of a period of a “High” level of each of the write signal I_WE, the write signal O_WE71, and the write signal O_WE72 is valid image data.
As illustrated in FIG. 7, transfer rates (cycles of the write signals O_WE71 and O_WE72) when the resizing units 71 and 72 output image data to the corresponding output DMA units 82 and 83 are lower than a transfer rate (a cycle of the write signal I_WE) when the distortion correcting unit 60 outputs image data to each of the resizing units 71 and 72. In addition, in the resizing units 71 and 72, transfer rates (cycles of the write signals O_WE71 and O_WE72) are different when image data after the resizing process is output to the corresponding output DMA units 82 and 83.
More specifically, as illustrated in FIG. 7, the resizing unit 71 having a predetermined resize ratio of ½ performs a resizing process of ½ at a ratio of one for two write signals I_WE, that is, a ratio of one for two lines, and transfers image data after the resizing process to the output DMA unit 82. In addition, the resizing unit 72 having a predetermined resize ratio of ¼ performs a resizing process of ¼ at a ratio of one to four write signals I_WE, that is, a ratio of one to four lines, and transfers image data after the resizing process to the output DMA unit 83.
This results from a resize ratio when each of the resizing units 71 and 72 performs the resizing process, and is because each of the resize operation processing units 714 provided in the resizing units 71 and 72 performs the resizing process according to the predetermined resize ratio after image data necessary to perform the resizing process has been aligned. Because of this, the resizing units 71 and 72 wait for an operation of the resize operation processing unit 714 until image data necessary to perform the resizing process is aligned within the input buffer 712, that is, there is a period in which the resize operation processing unit 714 does not operate in each of the resizing units 71 and 72 as indicated by “A” of FIG. 7. As a result, the activity rate of the resize operation processing unit 714 is lowered. In addition, a period in which the resize operation processing unit 714 does not operate becomes a period (see “B” of FIG. 7) in which no transmission of image data after the resizing process is performed.
Thus, in the image processing apparatus 100 of the related art, a necessary number of pipelines having image processing circuits with the same configuration are configured according to the number of processed images to be generated. However, if there is a period in which no image processing circuit operates as indicated by “A” in FIG. 7, there is a period in which the image processing circuit provided in each pipeline does not perform data transmission as indicated by “B” in FIG. 7.